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A Guid9 to Strategically Planning Training & Measuring Results 



"The way work is organized is being affected by the speed of 
change. Work processes are increasingly driven by what 
employees know — that is to say, how the work is done is 
increasingly dependent upon the level of knowledge the 
employee brings to the job. The more knowledgeable an 
employee is across disciplines, the better job she or he can do, 
and the more valuable she or he becomes." 

"The result of this trend is that the distinction between working 
and learning is becoming blurred — part of every employee’s job 
will be to keep learning about the ever-changing work to be 
performed." 



Janice Lachance 

Director, U.S. Office of Personnel Management 

Human Resource Training Forum (August 1999) 

Baltimore, Maryland 
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Making a Strategic Investment in Training: 

Executive Overview 

Chances are that you have read something lately or participated in a 
discussion about the payoffs of investing in training. For example, two 
major corporations recently made front-page news by providing their 
employees with home computers. These corporations believe that this 
investment will contribute to an acceleration of skills for both employees 
and the company throughout the 21st century. 

You are also aware that, in both public and private sectors, there are a 
growing number of examples of how a better trained workforce correlates 
with reduced costs, increased profitability, improved services, and 
increased customer satisfaction. No major, successful corporation fails 
to invest significantly in its people - the most important resource. 

According to data collected by organizations such as the American 
Society for Training and Development (ASTD) about leading-edge 
companies, one of the components that gives these companies an edge 
is commitment to and investment in learning. Further, research also 
shows that high impact companies, public-sector agencies, and other 
institutions attribute much of their front-runner status to recognition of the 
importance of including training as a key element in their strategic 
planning process. 

Throughout the business world, there is increasing recognition that 
training the workforce is a win-win business strategy. 



Workforce 

Investments 



The President of the United States provided pivotal direction to 
Government leaders about workforce learning and development in 
January 1999 through Executive Order 13111, Using Technology to 
Improve Training Opportunities for Federal Government Employees. 

This Executive Order embodies many tenets that Federal executives can 
use to harness the power of training. For example, learning should be an 
integral, planned part of doing business, as opposed to an afterthought. 
Every agency’s strategic plan should identify training and education as a 
means for achieving goals. It is a business strategy that makes sense. 

The Executive Order specifically calls on agencies to include a set of 
goals to provide effective training opportunities and allied performance 
measures as part of the annual budget process. 

Further, agencies are to identify the resources needed to achieve the 
accomplishment of those goals in their annual performance plans. This 
planning process is an important step in integrating and linking learning 
and education to accomplish organizational objectives. 




Executive Order 
13111 
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Making a Strategic Investment in Training: 

Executive Overview 



Typical Scenario 



The Challenges 



The Approach 



Imagine the following scenario. 

You and your planning team have 
established a performance goal to 
improve the efficiency of processing time 
for one of your agency’s mission-critical 
functions. 




Three months have now passed and the 
actual results are lagging behind the 
projected results. 



When analyzing the risks to goal achievement, the planning team 
identified a number of concerns including: 



■ Lack of computer capacity and other information systems. 

■ High turnover of key technical personnel due to competition from 
private industry. 

■ Loss of experienced employees due to a “retirement bubble.” 

■ Rapid evolving technology requiring staff to learn new skills. 

■ Inadequate public awareness of agency services. 



What is the key to 
addressing these risks 
to goal achievement? 




With the exception of the first item above, 
training could be a key element to ensuring goal 
achievement. Effective training can help provide 
the employees with the skills they need while 
addressing other human resource problems 
such as turnover. The key is to plan your 
training strategically. 



In most cases training can play a pivotal role in ensuring that your 
agency’s strategic and performance goals are met. The following 
process can be used to plan training strategically. 




♦ 
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Process Overview 



Making a Strategic Investment in Training: 

Executive Overview 

Listed below are the questions to ask when cornpleting each step. The 
subsequent chapters of this guide are designed to provide additional 
information, tips, and examples for those who are responsible for 
completing the steps. 



Analyze Established ■ To reach each goal, what competencies must the current and/or 
Goals future workforce members possess? 




Develop 

Training Strategies 




Integrate Training 
Into Strategic Plans 




Training Goal 
Accomplishments 



What benchmarks can be used to create innovative approaches to 
reaching this goal? 

Are there competency gaps that must be addressed to meet this 
goal? 

Could training help reduce other human resource problems such as 
high staff turnover? 



Could training address the competency gaps? 

Are there non-training strategies that are needed to support the 
training intervention? 

What types of training should be provided (e.g., classroom, distance 
learning, electronic performance support, on-the-job training, etc.)? 

Do the anticipated benefits from training outweigh the projected 
costs? 



What is the goal of the proposed training? 

Can you relate this training goal to an existing goal in your agency’s 
strategic plan? 

How will you measure training goal accomplishments? 

What tasks and resources are required to implement the training? 



Did you achieve the training goal? 

How much did it cost to achieve this goal? 




Did accomplishing this training goal help support the agency’s 
achievement of the related strategic goal? 

Did the benefits outweigh the costs? 

What modifications should we make to the strategic plan based on 
the evaluation findings? 
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Making a Strategic Investment in Training: 

Executive Overview 



The Results 



The Benefits 




The agency succeeded in reaching the 
strategic goal. 

To reach these results, the team analyzed 
the gaps and identified the non-training 
and training issues. 



■ Non-training Issues: To address the computer and systems capacity 
issue, the planning team conducted an extensive benchmarking of 
current system capability. As a result the team developed an 
updated Information Resource Management plan to address system 
shortfalls. 

■ Training Issues: The team established a business case that 
demonstrated the benefits of providing enhanced training and 
electronic performance support to new and current employees. 

Given the competencies of the existing workforce, the team found 
that it was more cost effective to contract with a public relations firm 
to develop the public awareness. 



Projected / 

Results'^-'^' Actual 
. • " ‘ Results 



The team then measured the interim and long-term results. The 
following training results support the achievement of the agency’s 
strategic goal: 

■ Increased skill level on use of automated systems. 

■ Increased knowledge of agency regulations. 

■ Increased effectiveness and efficiency in processing public inquiries. 

■ Decreased absenteeism and employee turnover due to the 
acquisition of new ergonomic and stress reduction techniques. 

■ Increased favorable customer service ratings. 



Purpose of This 
Guide 



Making a Strategic Investment in Training: 

Executive Overview 

This Guide is about planning training strategically. It was developed to 
assist agencies in fulfilling the direction contained in the Executive Order 
— to set goals and develop performance measures for training. It is a 
resource for those who plan the allocation and use of training resources, 
especially under the auspices of the Government Performance and 
Results Act (GPRA). 



In this document, you will find approaches to integrating training into the strategic 
plan, situations in which training can contribute to the accomplishment of strategic 
goals, and ideas about how to develop a business case for training strategies. 

Think of this Guide as a primer — a first step — to help Federal agencies begin to 
create long-term workforce investment strategies and plans. 



A wide agency audience — from managers, to human resource 
development professionals, to agency planners — can use this Guide. 

The ideas, examples, techniques and illustrations captured in this 
document will be of value to any agency, regardless of its particular 
strategic planning approach. These illustrations have come from both 
public and private sector entities that have recognized and successfully 
integrated strategic thinking into the ways in which training is used to 
achieve organizational goals. Some approaches will foster new and 
enduring collaborations among organizational components, as 
managers, budget analysts, program managers, and stakeholders focus 
on the potential of training as a key element in achieving corporate goals. 

Consider this Guide as a storehouse of ideas and tools that will help 
agencies achieve their goals and accomplish their mission. These tips 
and techniques can serve as an asset to people throughout the 
organization and can be useful at many junctures throughout the 
strategic planning and implementation process. Examples and 
techniques are offered as catalysts, not as mandates, to effectively 
consider and integrate training strategies into strategic and performance 
plans. 

How To Use This The remainder of this guide provides how-to guidance. The following 

Guide icons are used to signal the type of information being presented: 

This icon indicates that a “how-to" tip is being presented. 



This icon indicates that an example is being presented. 



This icon indicates that suggested additional resources are 
being presented. 
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Once, land and capital were the key strategic 
resources. Now, knowledge is our key strategic 
resource and learning is our key strategic skill. 

Vice President A1 Gore 
Lifelong Learning Summit Ostnuary 1 999) 
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Introduction 



Step 1: Analyze Established Goals To Identify 

Training Requirements 

The first step is to identify the training requirements based on your 
agency’s strategic and performance plans. As illustrated below, the 
strategic plan and annual performance plans contain blueprints for 
transforming the agency’s current performance into the desired future 
state. The performance plans include incremental goals that lead toward 
the achievement of the longer-term strategic goals. The performance 
plans are tools that can be used to help agencies identify the human, 
information/technological, and other resources to achieve the desired 
future performance. 

Training is an essential component of this approach. The training 
process ensures that any competency gaps within the workforce are 
addressed so that the agency can achieve the future performance goals. 




The Process 




Suggested activities for completing this step include: 

1 . Analyze the agency’s strategic and performance goals. 

2. Identify the enterprise (organization)-wide competencies that the 
workforce will need to accomplish these goals. 

3. Analyze gaps in the current or projected workforce. 

■ Required proficiencies and deficiencies 

■ Causes of deficiencies 




♦ 
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strategic Planning 
and Performance 
Accountability 



Step 1: Analyze Established Goals To Identify 

Training Requirements 

Congress' determination to make agencies accountable for their 
performance lay at the heart of two landmark reforms of the 1990s: the 
Chief Financial Officers (CFO) Act of 1990 and the Government 
Performance and Results Act of 1993 (GPRA). 

With these two laws, Congress imposed on Federal agencies a new and 
more businesslike framework for management and accountability. In 
addition, GPRA created requirements for agencies to generate the 
information congressional and executive branch decision makers need in 
considering measures to improve Government performance and reduce 
costs. 



Most fundamentally, under the Government Performance and Results Act of 1993 
(GPRA), every major Federal agency must now ask itself some basic questions: 
What is our mission? What are our goals and how will we achieve them? How 
can we measure our performance? How will we use that information to make 
improvements? GPRA forces a shift in the focus of Federal agencies — away from 
such traditional concerns as staffing and activity levels and toward a single 
overriding issue: results. GPRA requires agencies to set goals, measure 
performance, and report on their accomplishments. 

Charles A. Bowsher 
Comptroller General of the United States (June 1996) 



The Government Performance and Results Act (GPRA) is the primary 
legislative framework through which agencies will be required to set 
strategic goals, measure performance, and report on the degree to which 
goals were met. The Act requires each Federal agency to: 

■ Develop, no later than by the end of fiscal year 1997, strategic plans 
that cover a period of at least 5 years and include the agency's 
mission statement; identify the agency's long-term strategic goals; 
and describe how the agency intends to achieve those goals through 
its activities and through its human, information/technology, and other 
resources. Under GPRA, agency strategic plans are the starting 
point for agencies to set annual goals for programs and to measure 
the performance of the programs in achieving those goals. 

■ Submit to the Office of Management and Budget (0MB), beginning 
for fiscal year 1999, an annual performance plan. Agencies 
submitted the first annual performance plans in the fall of 1997. The 
annual performance plan provides a direct link between the strategic 
goals outlined in the agency's strategic plan and what managers and 
employees do day-to-day. 

■ Submit to the President and to the appropriate authorization and 
appropriations committees of Congress an annual report on program 
performance for the previous fiscal year. 
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GRPA and Training 



Step 1: Analyze Established Goals To Identify 

Training Requirements 

The following table summarizes the GPRA requirements and 
opportunities where agencies can strategically plan for training's role in 
supporting goal accomplishment. 



Activity 




Requirements 


Training Opportunity 


Develop and 
maintain strategic 
plans covering a 
5-year period 

Updated every 3 years 


Provide a description of: 

■ Agency's mission 

■ General goals and objectives 

■ The means and strategies to 
achieve goals and objectives 
(processes, skills, technologies, 
various resources) 

■ The relationship between 
performance goals in 
performance plan and general 
goals and objectives in strategic 
plan 

■ The key factors that can affect 
achievement 

■ The program evaluations to be 
used and schedule of 
evaluations 


Analyze strategic goals to 
determine short-term and 
long-term training 
requirements. 


Develop and 
maintain 

performance plans 

Updated every year 


Provide a description of: 

■ The relationship to strategic plan 

■ The performance goals and 
indicators (for each Program 
Activity in the budget) 

■ The operational processes, 
skills, and technology, and the 
human, capital, information, or 
other resources needed to meet 
goals 

■ The means to be used to verify 
and validate measured values 


As appropriate, create 
annual training 
performance goals to 
support the achievement of 
strategic goals. 

Use the performance 
measures when evaluating 
the value of training. 


Develop program 
performance reports 

Submitted annually 


Provide a description of: 

■ The relationship to strategic plan 
and performance plan 

■ The comparison of goals with 
achievements 

■ An explanation where goals 
were not met and plans for 
meeting unmet goals 


Report training 
performance goal 
accomplishments. 

Identify if training can help 
address any goals reported 
as unmet. 



• Tip: Your performance goals for training could be set in an 

evolutionary fashion. For example, the first-year goals could be to plan 
and design the training intervention, the next year to pilot test the 
intervention, and beyond the third year to implement and evaluate the 
intervention. 
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step 1: Analyze Established Goals To Identify 

Training Requirements 

After you have targeted the goals that could be enhanced, next you need 
to identify the enterprise (organization)-wide competencies required to 
support goal achievement. A competency is defined as a behavior or set 
of behaviors that describes required performance in a particular work 
context (e.g., job, role or group of jobs, function, or whole organization). 
Competencies can help ensure that individual and team performance 
aligns with the organization’s mission and strategic directions. 

At this point in the analysis, competencies are identified at the “macro 
level.” In a later step, you may need to repeat the process at the 
individual performer level. The following chart lists general competency 
areas you may want to assess: 



Competency Areas 


Questions To Consider 


Leadership 


■ What new skills will leaders need in the near term and longer 
term? 


Communication 


■ To achieve these goals, what types of communication 
competencies (interpersonal, written, oral) are required for 
working effectively within the organization and for meeting 
external stakeholder needs? 


Administrative and 
Work Processes 


■ Will new work processes be required to meet the goals? If 
so, what competencies are required to support these 
processes? 


Professional 


■ What specialized knowledge or skills (e.g., scientific, 

engineering, legal, medical, etc.) should performers possess? 


Analytic Problem 
Solving 


■ What new problem-solving challenges are associated with 
the achievement of the goals? What competencies will be 
needed to address these challenges? 


Decision Making 


■ Will regulatory changes require new approaches to decision 
making? 


Technology 


■ What types of technological competencies are required to 
reach these goals? 



Tip: Identifying 
Enterprise*Wide 
Competencies 






Step 1; Analyze Established Goals To Identify 

Training Requirements 



Tip: Identifying The most common methods to use in identifying the competency 

Enterprise-Wide requirements for goal achievement at the “macro level” are as follows: 

Competencies 
(Continued) 



Activities 


Description 


Review Existing 
Competency Models 


■ You may use existing competency models to produce a 
“strawperson” model or models as a starting point. 




The use of existing models coupled with additional data 
gathering decreases cost and improves the quality of the 
models. 


Conduct Focus 
Groups 


■ Next, focus groups or semi-structured interviews are conducted 
with selected groups of top performers and stakeholders to 
identify the competencies needed. 




It is critical that the data collection efforts be focused on 
the future competency requirements rather than today’s 
requirements. For the purpose of linking training to strategic 
planning, it may be necessary to include both near-team and 
longer-term competency expectations in your models. 


Validate Models 


■ Next, the competency models should be validated. Usually, 
validation is done through a survey and/or focus groups/ 
feedback sessions. 




To generate an initial approved version of models, the refined 
competency models are then presented to a stakeholder group 
for review and “sign off.” It is important to maintain a link 
between the competency model and the organization’s 
strategic and performance goals. 
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step 1: Analyze Established Goals To Identify 

Training Requirements 



Tip: Identifying 
Competency Gaps 




After you have identified the enterprise-wide competencies that are 
required for goal achievement, then you should assess to what degree 
the current and/or projected workforce possesses these competencies. 

As stated earlier, at this stage in the analysis you are analyzing 
performance at the “macro level” rather than for each individual 
performer. Even at the macro level, you may find it useful to conduct this 
analysis by organizational units or occupational series. 

The tools used to conduct a gap analysis include focus groups, surveys, 
and review of current performance data. In reviewing current 
performance data it is important to look at trends and project the data into 
the time span covered by the strategic and performance plans. Current 
performance data may not reveal future performance gaps. 



Tip: Determining 
Reasons for 
Competency Gaps 



After determining that there are competency gaps, it is important to 
determine the reasons for the gaps. The table below lists some common 
causes of competency gaps: 




Types of Causes 


Questions To Ask 


Environmental 

If you answer no . . . 

■ The environmental 
factors impacting 
performance must be 
addressed. 

■ Training alone is 
unlikely to address the 
performance gap. 


■ Will employees have adequate tools/technology, resources, or 
information to achieve the goals? 

■ Are job standards and expectations communicated in a timely 
and explicit fashion? 

■ Are job standards and expectations consistent with strategic 
and performance plans? 

■ Are employees satisfied with the general working conditions? 

■ Do work processes and rules (regulations) facilitate (rather 
than impede) goal attainment? 

■ Do standard operating procedures/work processes support the 
goals? 


Motivational 

If you answer no . . . 

■ Address motivational 
factors by aligning 
incentives with your 
strategic plan and 
removing disincentives 
for change. 

■ Training alone is 
unlikely to address the 
performance gap. 


■ Are there mechanisms in place to measure performance and 
provide feedback to performers? 

■ Will individual employees receive recognition for achieving the 
organization’s strategic and performance goals? 

■ Will there be consequences for poor performance that 
jeopardizes goal attainment? 

■ Is it unlikely that informal leaders will reinforce others for 
resisting change? 


Knowledge/Skills 

If you answer no . . . 

■ Lack of 

knowledge/skills can be 
solved through training. 


■ Have the employees successfully performed these 
competencies previously? 

■ Have employees been trained in the work processes and 
tools/technologies that will be used to reach goal attainment? 

■ Have managers and supervisors been trained to facilitate 
change management processes required to reach these 
goals? 




♦ 
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step 1: Analyze Established Goals To Identify 

Training Requirements 



The following example illustrates the process of identifying strategic 
training requirements: 




■ Ability to evaluate the effectiveness of outreach 
campaigns. 

■ Ability to build partnerships with States and 
other stakeholder groups. 

■ Knowledge of State occupant-protection-use 

laws. 

■ Ability to analyze data regarding enforcement 
best practices. 



Description of Gaps: 



* Employees can manage 
f ivde\fe\opment(>f public information campaigns 
using traditional media outlets but lack the 
skills associated with using the Internet for 
public awareness campaigns. 



■ Funding is needed for the development 
and use of Internet-based public 
awareness tools. 



Potential 
Strategic 
Training Need 



♦ 



Description of Gaps: 

■ Employees possess the analytic skills 
required to perform the analysis. The 
greatest performance barrier is the lack of 
incentives for State and local governments 
to share information. 

■ Employees need improved information 
technology resources to conduct analysis 
and communicate findings. 
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step 1: Analyze Established Goals To Identify 

Training Requirements 



Example 2 

e 
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The additional example again illustrates the process of identifying 
strategic training requirements: 



Strategic Goal: ^Maximize trade Compliance through a balanced 
program of informed«.compliance, targeted enforcement actions, 
^nd the facilitation otcomplying cargo. 



Performance Goal: Achieve a 95% compliance rate for all 
primary focus industries. 



Major Outputs and Tasks: 

1 . Detect areas of noncompliance through informed 
compliance and targeted enforcement actions. 

2. Raise compliance levels in primary focus industries. 

3. Get other nations to support compliance efforts. 



Enterprise-Wide Competency Requirements: 



Abjilty to apply a risk management approach to 
comptiance activities that is based on data analysis 
and targeting of industries for compliance monitoring. 



Ability to educate importers to encourage 
voluntary compliance activities and reporting. 



Ability to work with other nations to promote 
compliance activities and information exchange. 



Ability to use information technology to analyze 
trends and make projections. 

Ability to manage the change in compliance 
approaches/philosophies. 



Description of Gaps: 



Ail employees involved in importer compliance 
activities will need to be able to apply the new 
risk management approach. This process 
differs greatly from current practices. 



Potential 
Strategic 
Training Needs 



Description of Gaps: 

■ There is a need to expand the communication 
mechanisms used for the exchange of 
information with customs services in other 
nations. 



Staff members who interface with customs services 
internationally require additional skills In how to 
foster global partnerships. 
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Suggested 

Resources 




Step1: Analyze Established Goals To Identify 

Training Requirements 

The resources listed below provide additional information relating to the 
process of identifying strategic training requirements; 

Publications 

Dubois, D.D. Competency-Based Performance Improvement: A Strategy for 
Organizational Change. Amherst, MA: HRD Press, ^993. 

Gilbert, T. Human Competence: Engineering Worthy Performance. New 
York: McGraw-Hill, 1978. 

Harless, J.H. An Ounce of Analysis. Falls Church, VA: Harless Educational 
Technology, 1970. 

Jonassen, D.H., Tessmer, M., & Hannum, W.H. Task Analysis Methods for 
Instructional Design. Mahwah, NJ: Lawrence Erlbaum Associates, 1999. 

Mager, R.P. & Piper, P. Analyzing Performance Problems. Belmont, CA; 
Fearon, 1970. 

Spencer, L.M. & Spencer, S.M. Competence at Work: Models for Superior 
Performance. New York: John Wiley & Sons, Inc., 1993. 

U.S. Office of Personnel Management. Tra/n/ng A/eeds Assessment 
Handbook: A Guide for Conducting a Multi-Level Needs Assessment 
tHPDG-024;, January 1994. 

Web Resources 

http://www.epa.gov/epahrist/workforce/wap.pdf 

Workforce Assessment Project (Executive Summary and Tasks 1-4 
Final Reports), May 1999. This report documents ERA'S efforts to identify 
the critical skills needed today and into the next millennium to prepare the 
EPA workforce to meet the mission challenges. 

http://www.opm.gov/hrd/lead/FALLFEST/OPMBRI~1.HTM 

Cook, D. & Longanbach, P. A Competency-based Learning and 
Development Program, Federal Aviation Administration, The Center for 
Management Development, The Office of Training. 

This guide provides some definitions commonly used in a competency- 
based program. It lists validated competencies that are required for 
successfully accomplishing your organizational mission, and it shows, 
examples of the inventory instruments that are part of the program's 
model. The guide describes processes for answering the following three 
important questions: 

1 . What general and technical competencies does it take to effectively 
perform our jobs? 

2. What specific critical and important competencies must our 
employees develop in order to achieve our business mission and 
goals? 

3. How can we optimally provide learning and development competence 
for employees to close performance gaps? 
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Senior managers must include in their strategic plan | 
investing in workforce education through expanded I 
company training and development programs in I 

order to improve performance and productivity. I 

Corporate University Review 
(August 1999) 



♦ 

step 2: Develop Training Strategies 

To Achieve Goals 



o 

ERIC 




22 



BEST COPY AVAILABLE 



step 2 : Develop Training Strategies To Achieve Goals 



Introduction 



After identifying strategic training requirements, the next step is to 
explore alternative training strategies and to build a business case. 

As illustrated below, competency gaps are addressed using both training 
and nontraining solutions. If training solutions are to be pursued, then 
you first should weigh the costs and benefits of various training strategies 
and then build a business case to support the selected training strategy. 




The Process 



Suggested activities for completing this step include: 




1. Generate alternative training strategies for addressing competency 
gaps. 

2. Assess the costs and benefits. 

3. Build the business case for the selected approach. 



step 2: Develop Training Strategies To Achieve Goals 



There are many ways to provide training. The first decision to make is 
whether training will occur in a classroom, through distance learning, or 
in the workplace. 

The flowchart below lists the questions to ask when deciding if classroom 
training is appropriate. 





Tip: Generating 
Alternative 
Training Strategies 
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step 2: Develop Training Strategies To Achieve Goals 



Prior to making a commitment to providing training, it is important that 
you fully understand the potential costs and benefits. The table below 
lists the types of training costs that must be considered. 



Training Costs 



■ Development costs (e.g., salaries and benefits of personnel, equipment) 

■ Direct implementation costs (e.g., training materials, technology costs, facilities, 
travel, equipment, instructor’s salary and benefits) 

■ Indirect implementation costs (e.g., overhead. General & Administrative) 

■ Compensation for participants 

■ Lost productivity or costs of “backfilling” positions during training 

For additional guidance on calculating training costs, see U.S. Office of Personnel 
Management’s Making the “Business Case" for HRD Investments (HRDG-025), 
January 1994. 



As shown below, training benefits include increased outputs, resource 
savings, improved quality, and error reduction. 



Training Benefits 


Potential Benefits 


Indicators 


Increased Outputs 


■ Number of products produced or services provided 

■ Number of work processes completed 

■ Amount of backlogged work 


Time/Resource Savings 


■ Ratio of productive/non-productive time 

■ Amount of overtime required 

■ Amount of “break-in” time for new employees 

■ Amount of equipment “down time" 


Improved Quality 


■ Ratio of positive/negative customer feedback 

■ Ratio of positive/negative findings from internal 
audits/studies 

■ Amount of “rework" required 

■ Percentage of products/services meeting standards 

■ Number of innovative solutions/products developed 

■ Changes in employee morale 

■ Number of grievances and other personnel issues 


Error Reduction 


■ Number of errors 

■ Number of safety-related complaints 

■ Number of accidents 

■ Number of rule/procedure violations 

■ Number of products/services rejected 



Tip: Weighing 
Costs and Benefits 



O 
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step 2: Develop Training Strategies To Achieve Goals 



Tip: Building the After selecting the projected type of benefit and indicator, you should 

Business Case answer the following questions; 






Page 24 



Example 

e 



o 

ERIC 



step 2: Develop Training Strategies To Achieve Goals 

The following example illustrates the process of developing a business 
case for training: 
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Step 2: Develop Training Strategies To Achieve Goals 



Suggested 

Resources 




The resources listed below provide additional information relating to the 
process of developing a strategic business case for training. 

Publications 

Fisher, S.G. The Manager's Pocket Guide to Performance Management, 
Amherst, MA: HRD Press, 1997. 

Van Adelsberg, D. & Trolley, E.A. Running Training Like a Business: 
Delivering Unmistakable Value. San Francisco: Berrett-Koehler Publishers, 
Inc., 1999. 

U.S. Office of Personnel Management. Making the “Business Case” for HRD 
Investments: A Tool for Making Training and Development Decisions 
(HRDG-025), January ^994. 

Web Resources 

http://www. acq. osd. mil/dau/files/impiplan. pdf 

The Defense Acquisition University’s Technology-Based Training 
Implementation Plan provides an excellent example of building a 
business case for the use of technology-based distributed learning 
courses. 

http://www.opm.g 0 v/hrd/lead/events/atasofle:htmUtop 

A Taste of Learning: Tips, Tools, and Techniques for Developing the 
Workforce was held on September 29, 1999. Designed for the Federal 
community, A Taste of Learning is part.of a continuing series of 
leadership education seminars sponsored by the Office of Personnel 
Management's (OPM’s) Office of Human Resource Development. This 
link provides access to keynote addresses by Janice Lachance, 0PM 
Director, and Dr. Henry Kelly, then Assistant Director for Technology, , 
White House Office of Science and Technology Policy. Also available are 
handout materials from workshops offered on a variety of topics including 
diversity as a business strategy, competency modeling; distributed 
learning, and private-sector approaches to organizational measurement 
and evaluation. 

http://www.opm.gov/hrd/lead/leadindx.htm 

The U.S. Office of Personnel Management HRD Leadership Site 

provides online publications, policy guidance, training information, an 
events calendar, other resources, and links. 
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The American Society for Training and Development (ASTD) 
surveyed 540 U.S. corporations. The survey results, although 
not conclusive, indicated that there was a solid relationship 
between a company’s performance and its workplace learning 
and development practices. Companies that used innovative 
training practices were more likely to have improved 
performance over time. 

American Society for Training and Development 

(1998) i 
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step 3: Integrate Training Into Strategic Plans 



Introduction 



The purpose of this step is to incorporate the training strategy from the 
business case into the organization’s strategic plan. When training is 
integrated into the organization’s strategic plan, it is more likely to receive 
the required support and resources. 

As illustrated below, the strategic plan translates the organization’s 
mission into a 5-year plan. The strategic plan is the basis for the 
performance plan, which is a more detailed implementation blueprint. It 
is most likely that training goals will be added to the performance plan to 
support performance goals. In some cases, if the organization’s mission 
includes a training and education focus, then training goals may be 
included directly in the strategic plan. Whether the training goals are 
incorporated into the strategic or performance plans, they should include 
measures. 





Suggested activities for completing this step include: 

1. Write training goals. 

2. Identify performance measures (indicators). 

3. Develop an action plan. 



Tip: Writing 
Training Goals 




Step 3: Integrate Training Into Strategic Plans 

To be consistent with the Government Performance and Results Act 
(GPRA) guidance, your training goals should include target levels and 
performance indicators. 

Listed below are definitions and examples of the components that should 
be included in your training goals. (Source: This information was 
adapted from OMB’s Primer on Performance Measurement, February 28, 
1995.) 



Component 




Definition 


Example 


Performance Goal 
(Target Level) 


GPRA defines a performance 
goal as a target level of 
performance expressed as a 
tangible, measurable objective, 
against which actual performance 
can be compared, including a 
goal expressed as a quantitative 
standard, value, or rate. 


Improve maternal and child health on 
tribal reservations to meet 95 percent 
of the national standards for healthy 
mothers and children by 2001. (Note 
that this goal would rely on 
performance indicators (see below) to 
be measured effectively.) 


Performance 

Indicator 


GPRA defines a performance 
indicator as a particular value or 
characteristic used to measure 
output or outcome. 

Performance goals that 
incorporate performance 
measures into the target goal 
statements do not require 
separate indicators. 


Indicators for the maternal and child 
health goal above might include 
morbidity and mortality rates for this 
population, median infant birth 
weights, percentages of tribal children 
receiving full immunization shot 
series, frequency of pediatric 
checkups, etc. 



Tip: Identifying 
Performance 
Measures 
(Indicators) 




Different types of performance indicators can be used to measure goal 
attainment. The tables on the following pages describe the different 
types of indicators and provide examples of each. 

(Source: This information was adapted from OMB’s Primer on 
Performance Measurement, February 28, 1995.) 
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Step 3: Integrate Training Into Strategic Plans 


Indicator Type 


Description 


Outcome 


An outcome goal is an assessment of the results of a program compared to its 

intended purpose. 

Characteristics: 

■ Outcome goal attainment cannot be measured until a program (of fixed 
duration) is completed, or until a program (which is continuing indefinitely) 
has reached a point of maturity or steady state operations. 

■ While the preferred measure, outcomes are difficult to measure on an annual 
basis. (For example, an outcome goal setting a target of “by 2005, collecting 
94 percent of all income taxes annually owed" cannot be measured, as an 
outcome, until that year.) The measurement of incremental progress toward 
a specific outcome goal is sometimes referred to as an intermediate outcome. 
(Using the example above, a target of collecting 88 percent of taxes owed in 
2002 might be characterized as an intermediate outcome.) 

Example: 

40 percent of welfare recipients receiving job training are employed 3 months 

after receiving job training. 






Indicator Type 


Description 


Output 


An output goal sets a target level for the goods and services produced by a 
program or organization and provided to the public or to other programs or 
organizations. In GPRA, both outcome and output measures are set out as 
performance goals or performance indicators. 

Characteristics: 

■ Output indicators include process measures (e.g., paperflow, consultation), 
attribute measures (e.g., timeliness, accuracy, customer satisfaction), and 
measures of efficiency or effectiveness. 

■ Output may be measured either as the total quantity of a good or service 
produced, or may be limited to those goods or services with certain attributes 
(e.g., number of timely and accurate benefit payments). 

■ All outputs can be measured annually or more frequently. The number of 
output measures will generally exceed the number of outcome measures. 

Example: 

Annually provide job training and job search assistance to 1 million welfare 
recipients within 2 months of their initial receipt of welfare assistance. 
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step 3: Integrate Training Into Strategic Plans 



Indicator Type Description 

Impact Impact indicators measure the direct or indirect effects or consequences resulting 

from achieving program goals. An example of an impact is the comparison of 
actual program outcomes with estimates of the outcomes that would have 
occurred in the absence of the program. 

Characteristics: 

■ Measuring program impact often is done by comparing program outcomes 
with estimates of the outcomes that would have occurred in the absence of 
the program. 

■ One example of measuring direct impact is to compare the outcome for a 
randomly assigned group receiving a service with the outcome for a randomly 
assigned group not receiving the service. 

■ If the impacts are central to the purpose of a program, these effects may be 
stated or included in the outcome measure itself. 

■ Impacts can be indirect, and some impacts are often factored into cost- 
benefit analyses. An outcome goal might be to complete construction of a 
large dam; the impact of the completed dam might be reduced incidence of 
damaging floods, additional acreage converted to agricultural use, increased 
storage of clean water supplies, etc. 

■ The measurement of impact is generally done through special comparison- 
type studies, and not simply by using data regularly collected through 
program information systems. 

Example: 

Job training increases the employment rate of welfare recipients from 30 percent 
(the employment level of comparable welfare recipients who did not receive job 
training) to 40 percent (the employment rate of those welfare recipients who did 
I receive job training). 



Indicator Type 



Input 



ERIC 



Description 



Input indicators are measures of what an agency or manager has available to 
implement the program or activity. Inputs can include; employees (FTE), 
funding, equipment or facilities, supplies on hand, goods or services received, 
work processes or rules. When calculating efficiency, input is defined as the 
resources used. 

Characteristics: 

■ Inputs used to produce particular outputs may be identified through cost 
accounting. In a less detailed correlation, significant input costs can be 
associated with outputs by charging them to the appropriate program budget 
account. 

■ Often, a physical or human resource base (e.g., land acreage, square 
footage of owned buildings, number of enrollees) at the start of the 
measurement period is characterized as an input. 



Example: 

Reduce by 10% the required annual appropriation. 
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Step 3: Integrate Training Into Strategic Plans 

The Government Accounting Office (GAO) reviewed agencies' GPRA 
performance measures and identified the following best practices. 
(Source: Executive Guide: Effectiveiy impiementing the Government 
Performance and Resuits Act (06/01/96, GAO/GGD-96-118)). 



Best Practice 



Description 



Demonstrate 

Results 



Performance measures should tell each organizational level how well it 
is achieving its goals. 

As simple as this principle may appear, it poses an especially difficult 
challenge for Federal managers, for whom the link between Federal efforts 
and desired outcomes is often difficult to establish and may not, in fact, be 
apparent for years. 

Research programs provide one example. So do many health and welfare 
programs that are delivered jointly with State and local governments and 
third-party service deliverers. 

Implications for Training Goals: Whenever possible, training goals should 
measure the organizational results (outcomes) being achieved by the course 
graduates rather than be limited to measuring the training outputs. Given the 
challenges associated with demonstrating results, it is essential for training 
goals to build on the existing efforts being implemented for measuring 
performance. In addition, training goals may need to include input/output 
measures (e.g., numbers of people trained, resources used/saved, amount of 
skill transfer resulting, etc.) to augment the outcome or result indicators. 



Best Practice 


Description 


Limited to the 
Vital Few 


The number of measures for each goal at a given organizational level 
should be limited to the vital few. 

Those vital few measures should cover the key performance dimensions that 
will enable an organization to assess accomplishments, make decisions, 
realign processes, and assign accountability. 

Organizations that seek to manage an excessive number of performance 
measures may risk creating a confusing excess of data that will obscure 
rather than clarify performance issues. Limiting the number of performance 
measures to the vital few at each organizational level will help ensure that the 
costs involved in collecting and analyzing the data do not become prohibitive. 

Implications for Training Goals: An alternative to setting separate training 
goals is to add one or two training indicators to each performance goal. 
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Tip: Following 
Best Practices 




Best Practice 


Description 


Link to 

Responsible 

Programs 


Performance measures should be linked directly to the offices that have 
responsibility for making programs work. 

A clear connection between performance measures and program offices 
helps to both reinforce accountability and ensure that, in their day-to-day 
activities, managers keep in mind the outcomes their organization is striving 
to achieve. 

This connection at the program office helps to lay the groundwork for 
accountability as measures advance through the agency. By helping to lay 
the groundwork for accountability, a connection between performance 
measures and program offices also provides a basis for determining the 
appropriate degree of operational authority for various organizational levels. 
Managers must have the authority and flexibility for achieving the results for 
which they are to be held accountable. 

Implications for Training Goals: Training programs that are designed to 
address the organization’s strategic goals do not succeed without cooperation 
and shared accountability with the program offices. Collaboration is an 
essential ingredient for success. 
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Example 

e 
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The following example illustrates the process of linking strategic goals to 
training indicators: 



^Strategic Goal: ^Maximize: trade compliance'through|a balanced 
Iprpgram' targeted enforcement actions, and 
|ihe:faci|itet|pij; 6| com , | t !■ .i | ^ - 

.if., A *• , i. 



Performance Goal; Achieve a 95% compliance rate for all primary focus 
industries. 



Training Outcome Indicator 

■ 10% increase in collection of 
import fees. 



Training Output Indicator 

■ All personnel will receive training 
by the end of the fiscal year. 



Training Impact Indicator 

■ Risk management training will 
increase compliance levels by 95% 
(the comparable collection level 
without risk management is 85%). 
The value of this increase is 
estimated at more than $2.5 million. 



Training Input Indicator 

■ 20% decrease in compliance 
resources used. 
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After training goals and performance measures are identified, you should 
develop an action plan for implementing and evaluating the training 
strategy or intervention that has been selected. You can use the 
Instructional Systems Design (ISD) model to help identify the tasks to be 
completed. 

An action plan is a detailed description of the steps used to implement 
the selected training strategy. The action plan describes who performs 
each step and when the step is scheduled for completion. The resources 
necessary to carry out each task should also be specified in the plan. 



Your action plan should include the following elements: 



Task 




Responsibility 


Resources 


Schedule 


1. Conduct Front-End 
Analysis 


Lead Instructional 
Design 


1 0 staff days 

$15,000 contract 
expense 


Begin: 06-15-00 
End: 07-15-00 



Tip: Developing an 
Action Plan 






After the plan is implemented, you should establish a tracking system to 
monitor its the plan execution and impact. For each scheduled 
milestone, compare the actual performance with the proposed 
performance level, and report the results. 

For all variances in scheduled performance, ask the following questions: 



Analyzing Action Plan Performance Variances 

■ How does the reported performance compare with previous performance? 

■ Is the performance/schedule variance likely to prevent goal achievement? 

■ Are external factors affecting performance? If so, what are these factors? 

■ Is the performance/schedule variance due to unrealistic initial projections/planning? 
If so, what adjustments should be made? 

■ What modifications should be made to the action plan? 

■ When will performance information be collected again? 
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step 3: Integrate Training Into Strategic Plans 



Suggested 

Resources 




The resources listed below provide additional information relating to the 
process of incorporating training into the strategic plan. 

Publications 

Van Adelsberg, D. & Trolley, E.A. Running Training Like a Business: 
Delivering Unmistakable Value. San Francisco: Berrett-Koehler Publishers, 
Inc., 1999. 

Publications available on the Government Performance and Results Act 
Multimedia Orientation and Toolkit http://www.opm.gov/gpra 

U.S. Government Accounting Office. Executive Guide: Effectively 
Implementing the Government Performance and Results Act (06/01/96, 
GAO/GGD-96-118). 

U.S. Office of Management and Budget. Primer on Performance 
Measurement, February 28, 1995. 

Web Resources 

http://www.npr.gov 

This web site describes the National Partnership for Reinventing 
Government, and provides an extensive selection of GPRA documents, 
including primary reports, the final initiative. Presidential directives (the 
text of Executive Orders), Presidential memoranda, congressional action, 
and Government Accounting Office and Office of Management and 
Budget guidance. 

http://www. opm. gov/perform/articles/095, htm 

U.S. Office of Personnel Management, Workforce Performance 
Newsletter, Using a Balanced Scorecard Approach to Measure 
Performance. 

Traditionally, many Federal agencies have measured their organizational 
performance by focusing on internal or process performance, looking at 
factors such as the number of full-time equivalents allotted, the number of 
programs controlled by the agency, or the size of the budget. In contrast, 
private-sector businesses usually focus on the financial measures of their 
bottom line. Alone, neither of these approaches provides the full 
perspective of an organization's performance. This article explains that 
by balancing internal and process measures with results and financial 
measures, managers get a more complete picture of their organization’s 
performance. 

http://www. opm. gov/perform/articles/1 999/pdfl 0. htm 

U.S. Office of Personnel Management, Performance Management 
Practitioner Series, A Handbook for Measuring Employee 
Performance: Aligning Employee Performance Plans with 
Organizational Goals 

This handbook describes a method for developing employee performance 
plans that are aligned with and support organizational goals using an 
eight-step process. 
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The key competitive difference [for agencies] in the 2V^ 
century will be people. All agencies need to take stock of their 
employees and assess future workforce needs. 



David Walker, Comptroller General 
National Academy of Public Administrators Conference 
As reported in Federal Times (September 1 999) 
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step 4: Evaluate Training Goal Accomplishments 



Introduction 



When training is successful it supports the organization’s attainment of its 
strategic goals. Training begins as a response to a need or opportunity 
in an organization. Evaluating training results brings training full circle to 
ensure that it is addressing the needs, problems, or opportunities it was 
originally intended to serve. 

In specific, evaluating training goal accomplishments involves 
determining what outcomes or results have been gained by the 
organization and whether, considering the costs of training, that gain was 
worth the expense. 



Training 

Goal(s) 

Achieved? 





Did the 
Benefits 
Exceed the 
Costs? 



Did Training 
Support the 
Strategic Goal 
Attainment? 



Evaluating training helps decision makers manage scarce resources. 
When decision makers know the training’s contribution to the 
organization, they can: 

■ Identify those training programs that should receive funding. 

■ Justify increased expenditures on training. 

When decision makers are unsure how training contributes to the 
agency’s mission, they may find it difficult to justify training expenditures 
and may have to allocate limited training resources based on a "best 
guess" approach. 



The Process 



Suggested activities for completing this step include: 




1 . Assess training goal achievement. 

2. Determine the cost of training. 

3. Establish the worth of training. 

4. Compare the benefits to the costs. 

5. Assess results. 
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step 4: Evaluate Training Goal Accomplishments 



The starting point for your evaluation is to assess if the training goals 
were attained. If your training goals included performance indicators 
(see Step 3), then the process involves collecting data. The following 
table summarizes the different types of data collection methods. 



Method 




Advantages 


Disadvantages 


Interview 


■ Provides for more subtle and 
indepth information than other 
data collection methods. 

■ Permits flexibility to probe 
ambiguous questions and 
responses. 

■ Can verify other sources of 
information. 


■ Requires complex interviewing 
skills. 

■ Involves the time of both the 
interviewer and the interviewee 
to complete. 

■ Involves considerable cost. 

■ Results frequently in hard-to- 
quantify information. 


Survey 


■ Is easy to administer. 

■ Enables results to be 
processed quickly and at 
relatively low cost. 

■ Permits easy quantification and 
generalization of results due to 
standardized instrument. 

■ Gives respondents more time 
to think about answers. 

■ Assures anonymity of 
responses. 


■ Results frequently in individual 
differences in interpretation of 
items and scales. 

■ Can result in information of 
questionable validity. 

■ Is inflexible because the 
respondent must answer the 
questions asked. 

■ Nonresponders may bias the 
end results. 


Analyze Work 
Samples/ 
Performance 
Sampling 


■ Provides direct data on the 
performers' actual work 
outputs. 

■ Is unobtrusive; analysis occurs 
without interrupting the work 
setting or routine. 


■ Can be costly and time 
consuming. 

■ Requires analyst to understand 
the work area well enough to 
judge the samples. 


Review Existing 
Monitoring and 
Reporting Data 


■ Provides direct data on outputs 
and inputs. 

■ Is an extremely cost-effective 
means of collecting detailed 
information. 


■ Existing data may not be 
measuring the same types of 
indicators included in the 
training goals. 

■ Validity and reliability of the 
data reporting systems may be 
insufficient. 



Tip: Assessing 
Training Goai 
Achievement 
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step 4: Evaluate Training Goal Accomplishments 



There has been a growing demand for accountability of training costs 
within the Federal Government in recent years. Decision makers realize 
how large the investment— both monetary and nonmonetary— in training 
and development has become, and they need to be assured of the value 
of training. They want to know which training programs work, and at 
what cost. 

To determine the cost of training requires that you identify all of the 
expenses associated with designing, developing, implementing, and 
evaluating the training program. After you determine the types of 
expenses that apply to your program, you can then calculate costs. 

The most common types of training costs are listed below. Note that this 
list is not exhaustive, and you may have unique costs not mentioned 
below. 



Training Costs 




Description 


Training Attendance 
Costs 


Attendance costs are the expenditures associated with: 

■ Participant's time spent in training 

■ Travel and per diem costs 

■ Lost productivity or cost of replacing the individual \while in 
training 


Instructor Costs 


Instructor costs are the expenditures associated with: 

■ Instructor's time spent in training 

■ Travel and per diem costs 

■ Lost productivity or cost of replacing the individual while in 
training 


Instructional Development 
Personnel Costs 


Instructional development personnel costs are the personnel 
expenditures during the design and development of the training 
materials. 


Additional Instructional 
Development Costs 


Additional instructional development costs are the 
nonpersonnel resources used during the design and 
development of the training materials. 


Facility, Materii 
Equipment Cos 


al, & 
its 


Facility, material, and equipment costs include direct costs 
associated with the delivery of the training. 



Tip: Determining 
the Cost of 
Training 




Detailed worksheets for calculating training costs can be found in the 
0PM publication titled Establishing the Value of Training: Resource 
Guide (HRDG-023), March 1994. 



step 4: Evaluate Training Goal Accomplishments 

You can establish the worth of the training by converting the change in 
the performance indicator into a dollar value representing its worth. Next, 
you should convert the change in benefits into a dollar amount. If this 
step is not completed, you may find it impossible to compare costs and 
benefits. By not converting the benefit to a dollar value, you would be 
comparing "apples to oranges." Therefore, if you cannot establish an 
"objective" cost figure for the benefits, you should use a "subjective 
value." 

Some benefits are easy to convert into dollar amounts. For example, if 
there was a reduction in inputs, you can simply subtract the amount of 
overtime spent before training from the amount of overtime spent after 
training. 

However, some benefits are more difficult to convert to dollar amounts. 
The following method establishes a "subjective" measure of the worth 
resulting from improved quality. This method assumes that the agency 
does not get full value unless personnel are performing at a high quality 
level. In addition, the assessment of "quality" is based on a subjective, 
1-to-10 rating. You should only use this method when you cannot 
determine a more objective measure of improved quality. 



Subjective Method for Establishing the Worth of improved Performance Quality 


step 


Calculation 


Example 


1 . Determine the employee's annual 
compensation (salary plus benefits). 




$ 50,000 


2. Determine percentage of job being 
trained. 




20% 


3. Multiply annual compensation by 
percentage of job being trained. 


annual compensation (line 1) 
times 

% of job being trained (line 2) 


$ 10,000 


4. On a 1-to-10 scale, rate performance 
quality level BEFORE training. 


1 = Low Quality 
5 = Moderate Quality 
10 = High Quality 


3 


5. Multiply the BEFORE rating by 10 to 
determine the quality percentage. 


quality rating (line 4) 
times 
10 


30% 


6. Multiply the dollars spent on that 
portion of the job by the quality. 


Dollars spent on portion of job (line 3) 
times 

quality percentage (line 5) 


$ 3,000 


7. On a 1-to-10 scale, rate quality level 
AFTER training. 


1 = Low Quality 
5 = Moderate Quality 
10 = High Quality 


8 


8. Multiply the AFTER quality rating by 
10 to determine the new quality 
percentage. 


quality rating (line 7) 
times 
10 


80% 


9. Multiply the dollars spent on that 
portion of the job by the quality 
percentage. 


dollars spent on portion of job (line 3) 
times 

quality percentage (line 8) 


$ 8,000 


1 0. Determine the difference between 
BEFORE and AFTER worth. 


value of AFTER (line 9) 
minus 

value of BEFORE (line 6) 


$ 5,000 



T 



♦ 






Tip: Establishing 
the Worth of 
Training 
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Tip: Comparing 
Costs and Benefits 
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Tip: Assessing 
Results 




O 
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step 4: Evaluate Training Goal Accomplishments 

You can use the following calculation to compare training costs and 
benefits: 



Benefits per person 


— 


$ 7,000 


Cost of training per person 


— 


$ 3,000 


Return per person 




$ 4,000 


Times number of people trained 


X 


100 


Total Return on Investment 




$400,000 



Depending on how you established the worth of your benefits, your 
return-on-investment amounts or ratios may only show the savings 
accrued during the first year. You may want to project the continued 
benefits to be derived from the training over time before calculating your 
return-on-investment ratio. 



It is admirable to get a return-on-investment from a training effort. 
However, not all training will result in a return-on-investment. And, even 
if the training saved $100,000, it is important to link how those savings 
enhanced the organization’s ability to achieve its mission. 

Therefore, you should ask key stakeholders: 



;What contributions do 
you think training mide to 
the organization’s ability 
to reach its strateg ic and 

performance goals? What dollar value would 

you place on training’s 
contribution in helping 
the agency accomplish its 
goals? 




Returning to an earlier example . 



If import compliance is increased to 95% and $2.5 
million additional fees are collected . . . 

Then what was training’s contribution in helping 
realize this increase in performance? 



BEST COPY AVAILABLE 
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Step 4: Evaluate Training Goal Accomplishments 



Tip: Coming Full 
Circle 




Your evaluation findings are not just historical records of what 
was accomplished. The information you collect during the 
evaluation process is used to help revise and update your 
agency’s strategic and performance plans. 

As illustrated in the model shown below, planning training 
strategically is a cyclical process. 
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step 4: Evaluate Training Goal Accomplishments 



Suggested 

Resources 




The resources listed below provide additional information relating to training 
evaluation process. Following is a small sample of the numerous resources 
available. 

Publications 

Kirkpatrick, D.L. Evaluating Training Programs: The Four Levels {2"'‘ 

Edition), Englewood, CO: Berrett-Koehler Publishers, 1998. 

Wholey J.S., Hatry H.P., & Newcomer K.E., eds. Handbook of Practical 
Program Evaluation. San Francisco: Jossey-Bass Publishers, 1994. 

U.S. Office of Personnel Management, Establishing the Value of Training: 
Resource Guide (HRDG-023), March 1994. 

Web Resources 

http://www. ehr. nsf gov/EHR/RE C/pubs/NSF9 7- 1 53/start.htm 

This link provides access to a National Science Foundation (NSF) 
publication titled User-Friendly Handbook for Mixed Method 
Evaluations. The assumption guiding this handbook is that a strong 
case can be made for using an approach that combines quantitative and 
qualitative elements in program evaluations. Although the focus of this 
handbook is on evaluating NSF projects for funded projects, the 
information directly applies to designing and conducting training outcome 
evaluations. 



http://www.gao.gov 

The U.S. General Accounting Office (GAO) web site lists select 
GAO Policy and Guidance Materials, including such guides as: 

■ Designing Evaluations. PEMD-10. 1.4. May 1991 . 

■ Content Analysis: A Methodology for Structuring and Analyzing 
Written Material. PEMD-10.3. 1. September 1, 1996. 

■ Prospective Evaluation Methods: The Prospective Evaluation 
Synthesis. PEMD-10. 1. 10. November 1990. 

■ Quantitative Data Analysis: An Introduction. PEMD-10. 1.11. June 
1992. 

■ Using Structured Interviewing Techniques. PEMD-10.1 .5. July 1991 . 



http://www. ed. uiuc. edu/spedAri/evaiwkshp. htm 



An online transcript of a workshop titled Planning and Constructing 
Performance-Based Evaluations conducted by Joseph S. Wholey and 
John A. McLaughlin. This workshop directly links performance-based 
evaluations to GPRA requirements. 



http://www. astd. org/CMSAemplatesAemplate_ 1.html ?articleid= 11016 
This link contains an article titled Was It the Training? by Jack J. Phillips. 
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Glossary 



Abilities 



Benchmark 



Business Case 



Chief Financial 
Officers (CFO) Act 
of 1990 



Competency 



Effectiveness 



Efficiency 



Focus Group 



Government 
Performance and 
Results Act 
(GPRA) of 1993 

Impact Measure 
(Indicator) 



Attitudes, characteristics, or attributes that a person needs to possess to 
perform a job task. 

Comparative standard for evaluating accomplishments against known 
exemplars of excellence. A benchmark is a targeted goal that is beyond 
current capabilities, but for which the organization is striving. 

A method for projecting and documenting the benefits to be gained as a result 
of investing resources in a training intervention. 

The CFO Act was designed to remedy decades of serious neglect in Federal 
financial management operations and reporting. It provided for chief financial 
officers in the 24 largest Federal departments and agencies, which together 
account for about 98 percent of the Government's gross budget authority. In 
1994, Congress followed up on the CFO Act with the Government 
Management Reform Act of 1994. The latter extended to all 24 CFO Act 
agencies the requirement, beginning with fiscal year 1996, to prepare and 
have audited financial statements for their entire operations. 

A behavior or set of behaviors that describes excellent performance in a 
particular work context (e.g., job, role or group of jobs, function, or whole 
organization). Competencies can help ensure that individual and team 
performance aligns with the organization’s mission and strategic directions. 

Effectiveness refers to the level of achievement of program goals and the 
results intended (as defined in strategic plans and in legislation). (Examples: 
"percentage of trainees employed 1 year after completing job training"; 
"percentage of compliance in filing tax returns"; "percentage of reduction in 
recreational boating accidents.") 

Efficiency is the ratio of inputs to outputs; traditionally expressed as the 
amount of input (total costs) per unit of output, sometimes referred to as unit 
costs. (Example: "cost per transaction") 

A facilitator-led group of 6 to 12 participants convened in order to gather 
information about training needs and/or the impact of training. 

The Government Performance and Results Act (GPRA) is the primary 
legislative framework through which agencies will be required to set strategic 
goals, measure performance, and report on the degree to which goals were 
met. 

A measure of the direct or indirect effects or consequences resulting from 
achieving program goals. An example of an impact is the comparison of 
actual program outcomes with estimates of the outcomes that would have 
occurred in the absence of the program. Measuring program impact often is 
done by comparing program outcomes with estimates of the outcomes that 
would have occurred in the absence of the program. If the impacts are 
central to the purpose of a program, these effects may be stated or included 
in the outcome measures themselves. The measurement of impact is 
generally done through special comparison-type studies, and not simply by 
using data regularly collected through program information systems. 




Glossary 



Indicator 

Inputs 



Input Measure 
(Indicator) 



Instructional 
Systems Design 
(ISD) 

Instructor Costs 



Kirkpatrick's 
Evaluation Model 

Knowledge 
Needs Analysis 

Outcomes 



Outcomes - End 



See Performance Indicator. 

The resources used in producing an output or outcome. They are usually 
expressed as amount of dollars or amount of employee time (such as hours 
or FTE). The amount of inputs is not the amount of incoming workload that all 
programs have in one form or another. For example, the number of cases 
that come into a program is not an input as the term is defined. The amount 
of inputs, together with the amount of outputs, is used to calculate productivity 
and efficiency. 

Measures of what an agency or manager has available to carry out the 
program or activity; i.e., achieve an outcome or output. These measures can 
include employees (FTE), funding, equipment or facilities, supplies on hand, 
goods or services received, work processes or rules. When calculating 
efficiency, input is defined as the resources used. Inputs used to produce 
particular outputs may be identified through cost accounting. In a less 
detailed correlation, significant input costs can be associated with outputs by 
charging them to the appropriate program budget account. 

A systems approach to analyzing, designing, developing, implementing, and 
evaluating training. Each phase of the ISD process provides information that 
feeds directly into the next phase. 

Instructor costs are the expenditures associated with instructor's time spent in 
training, travel to the training site, and lost productivity or cost of replacing the 
individual while in training. 

A four-level model of training evaluation that allows the measurement of 
different training outcomes including participant reactions, learning, on-the-job 
behaviors, and organizational results. 

The facts, concepts, and principles needed to perform a job task. 

A methodological process of collecting and analyzing information to determine 
specific gaps in performance or the lack of information necessary to do a job. 
(Also referred to as needs assessment or front-end analysis.) 

The results of a program activity compared to its intended purpose. Quality, 
customer satisfaction and effectiveness indicators may all be used to 
measure outcomes. It may sometimes be useful to distinguish "intermediate 
outcomes" from "end outcomes," especially when end outcomes cannot be 
determined for 2 or more years into the future. Programs may have indicators 
for each of these categories. 

End outcomes are the desired and (ultimate) results that are hoped to be 
achieved by the program activities. These results are directly related to the 
agency's mission. (Examples; illnesses prevented; clients whose status 
improved; reduced crime; clean air or clean water.) 
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Outcomes - 
Intermediate 



Outcome Measure 
(Indicator) 



Outputs 



Output Measure 
(Indicator) 



Performance 



Intermediate outcomes are those results that are expected to lead to the ends 
desired, but are not themselves "ends." In many programs, a progression or 
sequence of outcomes usually occurs. For example, an environmental 
protection program aimed at returning inland lakes to clean condition will likely 
involve a sequence of measurable outcomes. These outcomes include, in 
order: passage of State and local laws or technical assistance to businesses 
("number of State and local programs that passed such laws or that provided 
such technical assistance"); businesses then take advantage of the technical 
assistance ("number of businesses that requested and received technical 
assistance from the program"); those businesses changed their behavior 
("number of assisted businesses that took steps to reduce their hazardous 
waste"); hazardous waste material flowing into lakes was reduced ("amount of 
hazardous waste released by the particular industry"); water quality improves 
("pollutant counts"); the condition of living resources in the receiving waters, 
such as fish and mammals, improves; and finally, citizens are able to use the 
waters for recreation and business purposes ("number of days of restricted 
fishing"). Only the last two outcomes should be considered as end outcomes. 

An assessment of the results, effects, or impact of a program activity 
compared to its intended purpose. Outcome measurement cannot be done 
until the results expected from a program or activity have been defined. As 
such, an outcome is a statement of basic expectations, often grounded in a 
statute, directive, or other document. Outcome measurement also cannot be 
done until a program (of fixed duration) is completed or until a program (which 
is continuing indefinitely) has reached a point of maturity or steady state 
operations. While the preferred measure, outcomes are often not susceptible 
to annual measurement. Also, managers are more likely to manage primarily 
against outputs rather than outcomes. The measurement of incremental 
progress toward a specific outcome goal is sometimes referred to as an 
intermediate outcome. 

The products and services produced by a program or process and delivered 
to customers. Outputs result from internal activity or effort. Contrary to normal 
usage, GPRA uses the term "output" to cover both process and output. The 
amount of outputs, together with the amount of inputs, is used to calculate 
productivity and efficiency. Quality is another output indicator. 

A tabulation, calculation, or recording of activity or effort that can be 
expressed in a quantitative or qualitative manner. The GPRA definition of 
output measure is very broad, covering all performance measures except 
input, outcome, or impact measures. Thus it covers output, per se, as well as 
other measures. Strictly defined, output is the goods and services produced 
by a program or organization and provided to the public or to the other 
programs or organizations. Other measures include process measures (e.g., 
paperflow, consultation), attribute measures (e.g., timeliness, accuracy, 
customer satisfaction), and measures of efficiency or effectiveness. Output 
may be measured either as the total quantity of goods or services produced, 
or may be limited to those goods or services with certain attributes (e.g., 
number of timely and accurate benefit payments). Some output measures 
are developed and used independent of any outcome measure. All outputs 
can be measured annually or more frequently. The number of output 
measures will generally exceed the number of outcome measures. 

The accomplishment of actions that transform inputs (through a process) to 
outputs and outcomes. Inputs, process, outputs, and outcomes can therefore 
be considered the "performance continuum," that is, the progression of 
performance actions. 
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Performance Goal 



Performance 

Indicator 



Performance 

Measurement 



Performance 

Plans 

Process 

Productivity 

Program 
Program Activity 



A target level of performance expressed as a tangible, measurable objective, 
against which actual achievements can be compared, indicating a goal 
expressed as a quantitative standard, value, or role. Thus, these are the 
targets set by the program for specific reporting periods. A performance goal 
is a statement composed of two components: an indicator and a target. For 
example: "to increase the immunization rates for 2-year olds by 40 percent by 
2001," includes the indicator immunization rates and the target to increase 
rates by 40 percent by 2001. GPRA requires annual goals for each indicator, 
but goals can be set for shorter periods (e.g., quarterly) for internal 
management purposes. 

A particular value or characteristic used to measure outputs or outcomes. 
Indicators are signs that point to success or failure in performance and 
answer the question: "How will we know when we have been successful?" 
They refer to what specifically is to be measured for each aspect of 
performance; i.e., the specific numerical measurements that are to be made, 
such as the "number of customer complaints." Performance indicators are 
usually expressed in numbers or percentage form. The denominator used to 
form the percentage (or ratio) usually includes such things as cost, time, total 
universe size, ora historical record (e.g., compared to last year). 

Performance measurement refers to measuring the performance of a 
program, a function, or a process. The term "performance" covers the work 
an organization(s) performs in converting inputs to outputs to outcomes. 

GPRA requires agencies to develop and maintain performance plans. 
Performance plans are submitted annually. The performance plan describes: 
the relationship to the strategic plan; the performance goals and indicators 
(for each Program Activity in the budget); operational processes, skills, and 
technology; the people, capital, information, or other resources needed to 
meet goals; and the means to be used to verify and validate measured 
values. 

The activities and tasks that add value to inputs to create outputs and 
outcomes. 

Productivity is the ratio of outputs to inputs, traditionally expressed as the 
amount of outputs per number Of employees or amount of employee time. 
When factors other than direct labor are included in the input, the inputs are 
expressed in cost terms and a more comprehensive productivity rating is 
achieved. Example: "number of transactions per FTE or number of 
transactions per total costs." Productivity is also the ratio of outcomes to 
inputs; e.g., number of persons employed after training per unit of input (either 
labor hours or expenditure of funds); amount of dollars collected per unit of 
input. 

Grouping of activities directly supporting a major function of an agency. 

A specific activity or project as listed in the program and financing schedules 
of the annual budget of the United States Government. 
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Program 

Evaluation 



Program 

Performance 

Reports 



Quality 



Return-on- 

Investment 

Stakeholders 



Strategic Plan 



Target Population 



Task 



T raining 

Attendance Costs 



Training 

Evaluation 



An assessment, through objective measurement and systematic analysis, of 
the results, impact, or effects of a program or policy — ^the manner and extent 
to which Federal programs achieve intended objectives. Program evaluations 
also are frequently used for measurement of "unintended" results, good or 
bad, which were not explicitly included in the original statement of objectives 
or foreseen in the implementation design. Evaluation can, therefore, serve to 
validate or find error in the basic purposes and premises that underlay a 
program or policy. 

GRPA requires agencies to submit an annual program performance report. 
This report presents a comparison of goals with agency's achievements, an 
explanation where goals were not met, and plans for meeting any unmet 
goals. 

Quality refers to timeliness, accuracy, and conformance to requirements. 
Quality requirements may be used to measure several aspects of 
performance, such as outputs, process, intermediate outcomes, and in some 
cases, end outcomes. (Examples; "number or percentage of products 
delivered on time"; "cycle time of process"; "percentage of work corrected or 
cost of nonqualify"; "number or percentage of products completed accurately"; 
"number or percentage of products conforming to requirements.") 

A comparison of the costs of training to the savings or other financial benefits 
derived as a result of training. 

Organizations, groups, or outside managers with a vested interest in the 
efficiency of the operations or the success of the organization in delivering 
effective outputs and maintaining the viability of a program. 

GPRA requires agencies to develop and maintain strategic plans covering a 
5-year period. The plan is updated every 3 years. The elements included in 
the strategic plan are: the agency's mission, general goals and objectives, the 
means and strategies to achieve goals and objectives (processes, skills, 
technologies, various resources), the relationship between performance goals 
in the performance plan and general goals and objectives in the strategic 
plan, the key factors that can affect achievement, and the program 
evaluations to be used and schedule of evaluations. 

The group of people for whom a training program is intended, usually defined 
in terms of age, background, and ability. (Also referred to as target 
audience.) 

A unit of work activity or operation that forms a significant part of a job 
function/duty (i.e., a major component of a job). A task constitutes a logical 
and necessary step in a performance, and usually has a logical beginning and 
end. A subset of a job. 

Attendance costs are the expenditures associated with participant's time spent 
in training, travel to the training site, and lost productivity or cost of replacing the 
individual while in training. 

The process used to measure the demonstrated ability of individuals and units 
to accomplish specific training objectives. 
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Training Methods 



Any of the many instructional approaches or combinations of approaches to 
achieve learning such as classroom presentations, technology-based lessons, 
case-study exercises, etc. 



Training Objective 



A clearly communicated statement of the desired changes in the target 
population's skills, knowledge, or abilities. A training objective often includes 
a description of the "activity" to be demonstrated, the "conditions" under which 
the activity will be performed, and the "standards" forjudging if the activity has 
been performed at the desired level. 
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